Researchers warn of 'catastrophic overtraining' in Large Language Models

The researchers compared two versions of OLMo-1b: one pre-trained on 2.3 trillion tokens and another on 3 trillion tokens.